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In March 2020, the coronavirus pandemic and attending shift to remote schooling initiated a 
dramatic impact on student learning, an impact that state and district leaders feel a sense of 
urgency to understand and address. These leaders are accustomed using state and district test data 
to shed light on student achievement and growth. But without state test data last year, and 
justifiable concerns about the trustworthiness of state assessment scores in the 2020-2021 school 
year, state leaders may feel they are in an almost impossible bind. These conditions warrant an 
expanded indicator system. 


Opportunity to learn (OTL) data can be helpful in this regard, and education leaders therefore 
should focus on designing and implementing OTL data-collection strategies. There is a long tradition 
of collecting such data as part of surveys of educational conditions and as components of 
large-scale testing programs. These data have been important for understanding varying levels of 
educational resources, such as access to high-quality 
curricular and instructional materials, as well as 
opportunities to learn in safe and healthy contexts. When 
OTL-related questions are embedded in tests, they have difficult to accurately interpret 
the added benefit of providing valuable information for 
interpreting test results. Many argue OTL data are 
necessary in any year, but particularly so in 2020-2021. year of continued COVID-19 
Indeed, it will be exceedingly difficult to accurately 
interpret state test scores this year—a year of continued 
COVID-19 disruptions—in the absence of additional data additional data to properly 
to properly contextualize the results. 


Indeed, it will be exceedingly 
State test scores this year—a 
disruptions—in the absence of 


contextualize the results. 


WHAT IS OTL? 


Opportunity-to-learn is a more than 50 year-old concept that has evolved from a focus on whether 
students have had sufficient access to instruction or content linked to particular concepts, to a more 
robust conception regarding the conditions and resources provided to schools to enable students to 
succeed. Floden (2002), in a summary of OTL measures associated with international assessments, 
reported that Husen (1967, pp. 162-163) receives credit for one of the earliest definitions of OTL, 
which held that OTL refers to whether “students have had the opportunity to study a particular topic 
or learn how to solve a particular type of problem presented by the test” (cited in Floden, 2002, p. 
232). Debra P. v. Turlington, a 1981 case in Florida, provides one of the most famous applications of 
student-level OTL data. Here, the U.S. Court of Appeals ruled that if the Florida high school 
graduation test “covers material not taught the students, it is unfair and violates the Equal 
Protection and Due Process clauses of the United States Constitution” (644 F.2d 397 [1981], as cited 
in McDonnel, 1995). 


' Thanks very much to Damian Betebbener, Juan D'Brot, Ted Coladarci, Nathan Dadey, Chris Domaleski, and the members of 
CCSSO’s Technical Issues in Large Assessment Collaborative for helpful comments and suggestions. Of course, any 
remaining errors are my own. 
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OTL entered the policy realm in the early days of the ' 
standards-based reform movement, with “debates over The broader notion of OTL—the 


educational standards and whether schools had a resources provided to school 
responsibility to provide students with an adequate 


opportunity to learn before they could be held districts by states, and to schools 


accountable for meeting achievement standards. That by school districts—is the 

debate came to a head in 1994 with the passage of the : ; 5 

federal Goals 2000 legislation” (McDonnel, 1995, p. 306). appropriate conceptualization as 
This legislation resulted in a call for OTL-related policies we consider the disruptive effects 


that describe the resources the state and district ; 
policymakers are expected to provide if students and of COVID-19 on educational 
teachers are to meet the high expectations prescribed by opportunities. 

state content standards. 


The National Council on Education Standards and Testing (1992) used the term “school delivery 
standards” instead of OTL to push policymakers to recognize that this matter was as critical to 
standards-based reform as were content standards and assessments: “School delivery standards 
should provide a metric for determining whether a school ‘delivers’ to students the ‘opportunity to 
learn’ well the material in the content standards” (p. E-5, emphasis in the original). 


The broader notion of OTL—the resources provided to school districts by states, and to schools by 
school districts—is the appropriate conceptualization as we consider the disruptive effects of 
COVID-19 on educational opportunities. UCLA’s Institute for Democracy, Education, and Access (no 
date) concisely summarized this viewpoint: 


Opportunity to Learn (OTL) is a way of measuring and reporting whether students and 
teachers have access to the different ingredients that make up quality schools. The more 
OTL ingredients that are present in an individual school, school district, or even in schools 
across the state, the more opportunities students have to benefit from a high quality 
education. OTL standards provide a benchmark against which the opportunities that a 
school provides can be measured. (p. 2) 


WHY OTL? In short, OTL data will support 


There are at least two reasons to collect OTL data in more appropriate interpretations 
2020-2021 (Marion, Gonzales, Wiener, & Peltzman, 2020). 
First, even if the 2021 state assessment is administered 
somewhat normally in schools, making valid 
interpretations of the results will be a challenge (Keng, Boyer, & Marion, 2020). And where a critical 
mass of students test remotely, which is the expectation, these important inferences about student 
and school performance will be even more tenuous. State leaders need to understand the 
circumstances and opportunities facing students in order to properly interpret test results this year 
and beyond. For example, did a student answer a question incorrectly because they did not know 
the concept tested, did not receive instruction on that concept, or did not have the technology to 
perform their best on the test? OTL may reveal that students were instructed asyncroynously (which 
is considered less effective than in-person or synchronous instruction) on many of the concepts 


of test score data. 
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where a particular class or group of students performed poorly on the test. In short, OTL data will 
support more appropriate interpretations of test score data. 


Second, summative assessments, even in the best of conditions, do not provide enough information 
necessary for policymakers to understand students’ learning context. States and districts need a 
broad array of indicators, such as whether students have adequate access to devices and 
broadband, the type (e.g., synchronous/asynchronous) and amount of remote instruction available, 
and the level at which students are engaged, to surface any inequitable opportunities and, in turn, 
direct resources where they are most urgently needed. 


Many previous OTL data-collection efforts focused on student- and teacher-level variables, such as 
whether students were instructed on certain concepts, the extent to which students received 
regular feedback on their performance, or whether students had seen questions like those on the 
test previously. My colleagues Nathan Dadey and Damian Betebenner outlined a set of high-priority 
indicators to capture whether students’ learning experiences are remote or in-person. COVID-19 
disruptions have put educational inequities in stark relief, crystalizing the need for state leaders to 
collect data at multiple levels of the system: state, district, teacher, parent, and student. Therefore, 
state leaders should construct a robust OTL data collection system whether statewide standardized 
tests are administered in any form this year or not. States can use this occasion to install a long- 
term, systematic OTL data collection and reporting system. 


OTL DATA COLLECTION 


Designing for the long-term 


States should move forward as quickly as possible with _..it makes sense to try to design 
designing a long-term OTL system, knowing that any new : 
data-collection effort will be both time- and resource- a robust OTL data collection and 
intensive—particularly this year. This does not mean that reporting system that can be 

full implementation needs to occur this year, but it makes ; 

sense to try to design a robust OTL data collection and rolled out over time. 


reporting system that can be rolled out over time. 


Designing an OTL system is similar to other design activities. State leaders will need to work with 
their assessment and accountability specialists, technical advisors, and key stakeholders to attend to 
the following general steps. 


1. Purposes and uses: What are the purposes and uses for the specific OTL indicator? The first 
step in designing a comprehensive OTL reporting system is to clearly articulate the purposes 
and intended uses of the new data to be collected. For example, if a state were able to collect 
student-engagement data and then link those data to student test scores, the announced 
purpose of this data collection might be to better understand the relationship between student 
engagement and test performance. 


2. Decisions: What decisions do you want to make and/or how do you plan to use the 
information? For any indicator, state leaders should ensure that data are only collected when 
there is a clear and compelling purpose to do so. How will we use this new piece of 
information, and how will this use improve the decisions we make? Following the example 
above, state leaders might decide to use their growing understanding of the relationship 
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between engagement and test performance to allocate funding and/or professional 
development support to school and districts struggling to engage students. 


3. Data Collection: What data do you need to be able to support these decisions/uses? State 
leaders must carefully consider what data and in what form will be necessary to address the 
intended uses. State leaders must clarify how the data will be collected and estimate the 
difficulty involved in doing so. 


4. Analysis Plan: System designers must create an analytic plan to ensure the proposed data and 
analysis approach will have a chance of yielding results to serve the intended uses. 


uw 


. Consequences: What are the potential unintended negative consequences of collecting these 
data? State leaders must anticipate likely misinterpretations and misuses of OTL data and work 
to preempt such misuses. For example, some are justifiably concerned that reporting the the 
more challenging conditions experienced by some students could lead to lower expectations 
for these students. Therefore, state and district leaders must provide additional resources and 
interventions to help students found to be lacking in opportunity the chance to meet grade 
level expectations. 


6. Audit: State leaders should commission an audit of currently collected data to avoid 
duplication of effort and to create a more efficient overall system. States collect a lot of data 
throughout departments of education and in many other state agencies. Before launching new 
data collections, states should inventory the data they already collect to see if it fits with their 
OTL designs established above and can serve the intended purposes. 


Starting now 

Dadey and Betebenner (2020) recommended that, because state agencies will be challenged with 
many demands this year, states should prioritize collecting a limited amount of student-level data 
this year. For example, the amount of time students experienced face-to-face instruction each week 
as well as the amount of time each week the student experienced synchronous and asynchronous 
instruction would be high-priority indicators to understand the influence of the pandemic on 
student learning. 


However, Dadey and Betebenner were concerned that relying on weekly or daily records could be 
overwhelming this year, and they suggested considering more general information collected at the 
school level. In other words, instead of keeping track of the amount of face-to-face instruction for 
each student each week, teachers could report the approximate time the class experienced face-to- 
face instruction. Such data are less able to be linked to student-level test scores, but this 
disadvantage would be offset by the reduced burden on data collection. 


In spite of Dadey and Betebenner’s cautions, | maintain that system-level (e.g., device availability and 
internet access) data are necessary to understand the scope and inequities of the pandemic- 
induced disruptions. Understanding the effects of COVID-19 disruptions requires an understanding 
of what | call institutional OTL, which pertains to the resources and access provided by states and 
districts. Indicators include internet access, device availability, the presence of a high-quality 
curriculum, and teacher professional development to support remote learning, all of which help 
policymakers thoughtfully interpret test scores and then direct resources as needed. State leaders, 
therefore, should craft an OTL data-collection plan across multiple levels of the system and, in turn, 
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design reporting systems that allow stakeholders to appreciate the degree to which learning 
opportunities are available for all students. To move from the abstract to the concrete, | provide in 
Appendix A examples of indicators for different categories of stakeholders. | also describe various 
data-collection approaches and, for each, the expected difficulty of collecting the data. 


Continuous improvement and evaluation 

While it is tempting to collect any data on anything that 
one might find intellectually interesting, all data collection ensure that all data elements 
and reporting systems require resources and effort from 
both those collecting the data as well as those submitting 
the information. Therefore, it is critical to ensure that all indirectly, to student 
data elements are related, either directly or indirectly, to 
student achievement. As part of the analytic plan 
described above, states should design a continuous 
improvement and evaluation plan to regularly refine the OTL system. In addition to examining the 
quantitative relationships among the indicators and student achievement, states should evaluate 
the degree to which stakeholders use the OTL reports as intended. 


Therefore, it is critical to 
are related, either directly or 


achievement. 


Final thoughts 

Again, | recognize that state and local education professionals are overwhelmed this year; some 
readers may wonder if | am suggesting just one more thing. That is not my intention, which is why | 
propose that state leaders and key stakeholders approach this work deliberately and balance OTL 
data collection efforts against competing demands. States 

should strategically determine which data should be 

collected and reported this year to better understand the OTL information can help 
impact of the COVID-19 disruptions on students and leaders and educators 
schools, and when and how additional indicators will be 

subsequently phased in. OTL information can help leaders understand and improve 


and educators understand ane improve educational 7 educational opportunity, one 
opportunity, which is one step in bridging the equity divide. : om vie, 
step in bridging the equity divide. 
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APPENDIX A: 


EXAMPLES OF POTENTIAL OTL INDICATORS 
AND DATA COLLECTION APPROACHES 


Parent 
availability to 
support 
learning 


and prevailing 
conditions for 
remote learning 


* Student surveys 


from parent surveys, 
especially when asking 
about potentially sensitive 
issues. 


Lack of access to 
technology could inhibit 
the data collection for 
some respondents; 
multiple modes of 
feedback should be 
available as necessary, 
which can be challenging 


Questions of students can 
be included on tests or 
otherwise surveyed 
without tests 


Level of Potential Potential Data- 
Data : Collection Likely Effort Required Examples 
; Indicators 
Collection Approaches 
State/ * Internet * Maps of internet + Mapping internet service 1. Maps of internet service, 
District availability and access and will require cooperation including average 
capacity bandwidth across from ISPs bandwidth, produced by 
* Device the state * Device inventory should be census block. 
availability * Indpendent audit relatively straightforward 2. A roster of devices by 
* Presence and of mechniGloey * EdReports and other students by schools. 
access to, capacity and organizations provide 3. Audit report by 
high-quality suai independent curriculum appropriately 
curriculum * District device and | reviews, which should be credentialed, independent 
«Presencednd distribution easy to gather service provider 
access to IVE NLOLY * Local data-collection efforts | 4. Public reporting of specific 
high-quality * Independent can be difficult to curriculum used for ELA 
professional curriculum reviews | aggregate to the state level and math by school. 
development * Feedback surveys peeaise ually ee yer 5. Rates of reporting for 
regarding os be ie oe ideal. surveys, qualitative 
response, quality, oe : ae a ian bait statements regarding use 
and impact ss Saas coe a of high-quality curriculum, 
pe elaer eer a application of devices, or 
aslo) co ara and relevant use of internet or 
the means to deliver it. infeastruictire: 
Parent/ * Types of * Parent surveys + It will be challenging to get | 1. What type of space does 
Caregiver available spaces a reasonable response rate your student have to do 


school work and take 
tests? 


a. A quiet, private space 
with few distractions. 


b. A space shared with 
others and/or one 
that is not free from 
distractions 


2. Please describe your 
availability to support 
your student's learning 
and assessment. 


a. 1am often away from 
home while my 
student is often at 
home. 


b. | am occasionally 
available to help my 
student with learning 
and testing as 
appropriate. 


c. | am available almost 
full-time to assist my 
student with learning 
and testing as 
appropriate. 
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APPENDIX A (CONTINUED) 


including such 
indicators as 
the time 
students spend 
completing 
assignments? 


Instructional 
practices, such 
as the 
proportion of 
time spent in 
syncronous Vs. 
asynchronous 
instruction, the 
amount of time 
the teacher is 
able to interact 
directly with 
students 
(remote or 
in-person) 


Curriculum 
choices, such as 
prioritized 
standards 


Teacher 
professional 
learning 
opportunities 
related to 
remote/hybrid 
instruction 


* School schedules 
(e.g., in-person, 
remote, and 
hybrid days and 
times) 


* Modified surveys 
of the enacted 
curriculum‘? 


* Curriculum 
documents 


because school personnel 
are so overwhelmed this 
year just meeting students’ 
needs, state leaders will 
have to approach such data 
collection judiciously. 


Level of Potential Potential Data- 
Data : Collection Likely Effort Required Examples 
: Indicators 
Collection Approaches 
Teacher/ * Number of * Teacher logs and__| Many of these indicators 1. Engagement may be 
Schools? “meaningful attendance would not be difficult to operationalized as 
engagements,” records collect in normal times, but assignments completed, 


contact hours with 
students, or student-to- 
student interactions, 
among other things. 


2. Instructional practices 
may include 
documentation of how 
“classroom” routines were 
enacted (e.g., 
synchronously, 
asynchronously), the types 
of assignments provided 
to students, and how 
instruction was organized 
(e.g., whole group, 
individual work, small 
group). 


A potential survey item in 
this vein could be like the 
following: 


3. Please examine each test 
question in turn and 
indicate in the way 
described below, whether, 
in your opinion: 

a. All or most of this 
group of students 
have had an 
opportunity to learn 
this type of problem. 


b. Some of this group of 
students has had an 
opportunity to learn 
this type of problem. 


c. Few or none of this 
group of students has 
had an opportunity to 
learn this type of 
problem. 


2 See: Nathan Dadey & Damian Betebenner’s recent blog for many examples at student and teacher levels. 


3 While this is really a student-level indicator, it makes sense to collect the information from teachers. 


4 The Survey of the Enacted Curriculum is very time consuming to complete. Therefore, the only practical way to use such 
data collection approaches this year would be to either shorten the instrument considerable and/or to employ some type of 
matrix-sampling approach. 
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APPENDIX A (CONTINUED) 


* Effort on test 


* Remote spaces 
for school work 


sampled). But if assessments 
are not administered, it will 
more challenging to collect 
these data and it would not 
make sense to ask questions 
about the testing 
experience. Rather the focus 
of such surveys would be on 
classroom or remote 
learning experiences. 


Level of Potential Potential Data- 
Data : Collection Likely Effort Required Examples 
: Indicators 

Collection Approaches 

Students * Enacted/ * Stand-alone If assessments are Examples of the types of 
experienced student surveys, administered, it should be questions that need to be 
curriculum or appended to relatively easy to include collected with testing 

«Laval of assessments short student surveys on (Sessoms & Finney, 2015): 

engagement the tests (perhaps matrix- 1. Did you learn these 


concepts in class? 


2. How regularly did you see 
problems like this in class? 


3. Doing well on this test was 
important to me. 


4. | tried as hard on this test 
that | do on tests in my 
class. 


5.1 gave my best effort on 
this test. 


6. | felt more distracted 
when taking this test 
compared to tests | take in 
school. 


Examples of questions not 
reliant on aa testing 
situation: 


1. How often were you able 
to access online lessons? 


2. How well were your 
teachers able to meet 
your learning needs? 


3. What kind of space do you 
have for doing your 
remote school work? 
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